Papers with generalization of retrieval-augmented language models
On the Generalization Ability of Retrieval-Enhanced Transformers (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent work on retrieval-augmented language models has shown impressive results . performance gains from retrieval to a large extent originate from overlapping tokens between the database and test data, suggesting less of non-trivial generalization than previously assumed. |
| Approach: | They propose to off-load memory from trainable weights to a retrieval database and compare it to larger models with a larger model. |
| Outcome: | The proposed model outperforms GPT-3 and Jurassic-1 on the Pile at 4% of the model parameters. |